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SUBJECT: 


Technical  Assessment  Report  of  the  Precision  Measurement 
Equipment  Laboratory  Automated  Management  System 
(Report  95-004) 


This  report  is  provided  for  your  information  and  use.  We  conducted  the 
Technical  Assessment  in  response  to  a  Hotline  allegation  that  the  Air  Force  wa 
waSg  nS  ustag  Commerdal-Off-The-Shelf  products  to^usj.  Precision 

Measurement  Equipment  Laboratory  Automated  Management  System  (P.^S) 
requirements  PAMS  is  an  Air  Force-developed  computer  system  that  automates  the 
Smiiistrative  aspects  of  metrology  and  calibration.  Management  comments  on  a  draft 
ofTte  iSS  wercoiSideted  in  preparing  this  report.  Air  Force  consents  conformed 
to  die  requirements  of  DoD  Directive  7630.3  and  resoived  all  issues.  Therefore, 
additional  comments  are  not  required. 

The  courtesies  extended  to  the  Technical  Assessment  Division  „ 

annreciated  If  vou  have  questions  on  this  assessment,  please  contact  Mr.  Kenneth  H. 
appreciatea.  II  you  I  4  604-8952  (DSN  664-8952)  or  Mr.  Gregory  R. 

SXn  JSSan  ger  al  703)  60n94?(DSN  664-8946).  The  distribution  of 
Kpo«  is  iS  in  Appendix  D.  The  technical  assessment  team  members  are  listed 

inside  the  back  cover. 


David  K.  Steersman 
Deputy  Assistant  Inspector  General 
for  Auditing 


Office  of  the  Inspector  General,  DoD 


Report  No.  95-004  October  4,  1994 

(Project  No.  3PT-8016) 

TECHNICAL  ASSESSMENT  OF  THE  PRECISION  MEASUREMENT 
EQUIPMENT  LABORATORY  AUTOMATED  MANAGEMENT 

SYSTEM 


EXECUTIVE  SUMMARY 


Introduction.  The  Precision  Measurement  Equipment  Laboratory  Automated 
Management  System  (PAMS)  is  an  Air  Force-developed  computer  system  that 
automates  the  administrative  aspects  of  metrology  and  calibration. 

We  received  a  Hotline  complaint  asserting  that  the  Air  Force  has  wasted  resources  by 
continuing  to  use  PAMS  when  more  capable  Commercial-Off-The-Shelf  products  are 
available  at  lower  costs.  The  complainant  cited  three  specific  deficiencies  concerning 
the  development  of  PAMS  to  support  this  conclusion.  These  deficiencies  were: 

o  PAMS  was  not  developed  in  accordance  with  Air  Force  directives, 

o  the  Air  Force  was  planning  to  reverse-engineer  PAMS  and  convert  the 
software  from  the  Basic  computer  language  to  Ada,  and 

0  an  independent  validation  and  verification  of  PAMS  was  never  completed. 

The  complainant  believed  that  the  PAMS  software  would  be  difficult  and  expensive  to 
maintain  and  that  reverse-engineering  and  converting  the  PAMS  software  to  Ada  would 
be  very  costly.  The  complainant,  therefore,  concluded  that  Air  Force  could  save 
money  by  replacing  PAMS  with  a  Commercial-Off-The-Shelf  product. 

Objective.  The  objective  of  our  technical  assessment  was  to  determine  whether  the  Air 
Force  had  wasted  resources  by  using  PAMS  instead  of  a  Commercial-Off-The-Shelf 
product. 

Technical  Assessment  Results.  The  Air  Force  did  not  waste  resources  by  using 
PAMS  instead  of  a  Commercial-Off-The-Shelf  product  (Finding  A).  The  complainant 
correctly  identified  several  PAMS  deficiencies,  but  the  Air  Force  has  also  recognized 
these  deficiencies  and  corrected  them  for  the  most  part. 

As  a  result,  the  PAMS  software  has  not  been  overly  troublesome  or  expensive  to 
maintain.  Further,  the  PAMS  users  are  satisfied  with  the  system.  Consequently,  the 
complainant's  recommendation  that  PAMS  be  replaced  with  a  Commercial-Off-The- 
Shelf  product  could  not  be  supported. 

The  Air  Force,  however,  has  yet  to  correct  one  PAMS  problem  that  stems  from  the 
deficiencies  identified  by  the  complainant.  The  Aerospace  Guidance  and  Metrology 
Center  routinely  does  not  receive  PAMS  data  from  approximately 


40  percent  of  the  Precision  Measurement  Equipment  Laboratories  because  PAMS 
cannot  communicate  with  other  Air  Force  maintenance  management  systems 
(Finding  B).  PAMS  does  not  communicate  with  these  other  systems  as  the  result  of 
two  factors: 

o  PAMS  was  not  fully  integrated  into  a  Communications-Computer  Systems 
(C-CS)  architecture  when  the  system  was  developed,  and 

o  a  PAMS  Program  Management  Directive  was  not  implemented  when  the 
PAMS  program  management  responsibility  transferred  from  the  PAMS  Design  Office 
to  the  Standard  Systems  Center. 

Although  the  Air  Force  is  aware  of  the  PAMS  communications  problems,  it  has  not 
addressed  the  causes  of  these  problems.  Instead,  the  Aerospace  Guidance  and 
Metrology  Center  has  instituted  a  short-term  solution  that  b3q)asses  the  communications 
problems;  the  TMDE  data  are  being  mailed  to  the  Aerospace  Guidance  and  Metrology 
Center  on  computer  discs.  While  this  solution  is  expected  to  improve  the  percentage  of 
data  that  reaches  the  Aerospace  Guidance  and  Metrology  Center,  it  is  person-power 
intensive  and  an  inefficient  use  of  the  Air  Force's  computer  resources.  Two  additional 
reasons  warrant  the  timely  correction  of  the  PAMS  communications  problems.  First, 
the  Air  Force  will  pay  less  for  the  necessary  corrections  the  sooner  tot  the  problems 
are  addressed.  Second,  by  correcting  the  PAMS  communications  problems,  the  Air 
Force  will  lay  a  solid  foundation  for  future  PAMS  improvements,  whether  they  be 
upgrades  to  the  current  PAMS  configuration  or  Commercial-Off-The-Shelf  products. 

Summary  of  Recommendations.  We  made  no  recommendations  in  Finding  A.  In 
Finding  B,  we  recommended  tot  the  Air  Force  correct  the  PAMS  communications  and 
management  problems  by  fully  integrating  PAMS  into  a  Communications-Computer 
Systems  architecture  for  Air  Force  maintenance  management  systems  and  establishing  a 
PAMS  Program  Management  Directive. 

Management  Comments.  The  Air  Force  agreed  with  the  substance  of  the 
recommendations,  but  provided  an  alternative  approach  for  implementing  the 
recommendations.  The  Air  Force  agreed  tot  PAMS  functionality  should  be  integrated 
into  a  Communications-Computer  Systems  architecture  for  Air  Force  maintenance 
systems  and  tot  a  corresponding  Program  Management  Directive  is  warranted. 

Technical  Assessment  Comments.  The  Air  Force's  comments  were  responsive  and 
consistent  with  our  recommendations.  Consequently,  additional  comments  on  this  final 
report  are  not  required.  Further  information  on  management  comments  is  in  Part  II, 
and  the  complete  text  of  the  comments  is  in  Part  IV. 


*The  40  percent  value  is  based  on  an  input  of  38  percent  from  Air  Force 
representatives  familiar  with  the  problem  and  a  40  percent  value  calculated  from 
the  Aerospace  Guidance  and  Metrology  Center's  Maintenance  Data  Collection 
reports  for  the  month  of  March  1993  (zero  Test,  Measurement,  and  Diagnostic 
Equipment  (TMDE)  data  was  received  from  62  of  155  PMELs). 
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Background 


The  Air  Force  relies  on  precise  measurement  equipment  (oscilloscopes,  for 
example)  to  ensure  that  safety  and  mission-critical  instruments  (such  as 
altimeters  and  pressure  gauges)  are  properly  maintained.  The  precise 
measurement  equipment  are  more  commonly  termed  Test,  Measurement,  and 
Diagnostic  jEquipment  (TMDE)  and  are  themselves  sensitive  instruments  that 
must  be  periodically  calibrated  to  ensure  that  they  are  operating  properly. 

For  optimum  efficiency,  the  calibration  intervals  of  TMDE  must  be  closely 
monitored.  The  calibrating  of  TMDE  requires  that  the  equipment  be  removed 
from  service;  therefore,  the  calibration  periodicity  must  be  balanced  to  eliminate 
unnecessary  calibrations  while  still  ensuring  that  the  TMDE  remains  accurate. 
To  achieve  this  balance,  data  are  collected  whenever  a  piece  of  TMDE  is 
calibrated. 

A  computer  system  is  necessary  to  manage  the  Air  Force's  TMDE  data.  The 
Air  Force  utilizes  more  than  860,000  pieces  of  TMDE  and  data  must  be 
collected  on  each  piece  of  measurement  equipment.  The  data  are  used  for 
several  purposes,  including: 

o  tracking  the  maintenance  history  and  status  of  each  individual  piece  of 
measurement  equipment  and 

o  updating  the  TMDE  calibration  intervals. 

The  Air  Force  developed  the  Precision  Measurement  Equipment  Laboratory 
(PMEL)  Automated  Management  System  (PAMS)  to  effectively  collect, 
manage,  and  disseminate  TMDE  data.  Before  the  development  of  PAMS,  the 
Air  Force  manually  conducted  these  tasks;  however,  this  process  was  time- 
consuming  and  could  be  more  readily  accomplished  by  a  computer. 

We  received  a  Hotline  complaint  asserting  that  the  Air  Force  was  wasting 
resources  by  utilizing  PAMS  when  more  capable  Commercial-Off-The-Shelf 
products  are  available  at  lower  costs.  The  complainant  cited  the  following 
problems  with  PAMS: 

o  PAMS  was  not  developed  in  accordance  with  Air  Force  Regulations^; 

o  the  Air  Force  has  planned  to  reverse-engineer^  PAMS  and  convert  the 
software  from  the  Basic  computer  language  to  Ada;  and 


^The  700  series  of  the  Air  Force  Regulations  govern  the  development  of 
Automated  Data  Processing  systems,  such  as  PAMS. 

2ln  this  case,  reverse-engineer  means  taking  the  existing  PAMS  code  and 
working  backward  to  document  the  software's  structure. 
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o  an  independent  validation  and  verification^  has  never  been  conducted 
for  PAMS. 

The  complainant  believed  the  PAMS  software  would  be  difficult  and  expensive 
to  maintain  and  that  the  reverse-engineering  and  conversion  of  the  PAMS 
software  to  Ada  would  be  very  costly.  Consequently,  the  complainant 
concluded  that  the  Air  Force  could  save  money  by  replacing  PAMS  with  a 
Commercial-Off-The-Shelf  product. 


Objectives 


The  overall  objective  of  this  technical  assessment  was  to  determine  whether  the 
Air  Force  was  wasting  resources  by  using  PAMS  instead  of  a  Commercial-Off- 
The-Shelf  product.  More  precisely,  our  objectives  were  to  determine  the 
validity  of  each  specific  problem  cited  by  the  complainant;  whether  the  Air 
Force  had  corrected  these  problems;  and  whether  these  problems  resulted  in 
wasted  resources  that  could  have  been  saved  by  replacing  PAMS  with  a 
Commercial-Off-The-Shelf  product. 


Scope  and  Methodology 


We  traced  the  chronological  history  of  the  specific  problems  cited  by  the 
complainant.  We  first  evaluated  whether  the  complainant  had  accurately 
identified  the  PAMS  problems.  We  assessed  the  actions  taken  by  the  Air  Force 
to  correct  the  problems  and  analyzed  whether  the  problems  resulted  in  wasted 
Air  Force  resources.  We  also  reviewed  several  potential  PAMS  improvement 
options,  including  the  replacement  of  PAMS  with  a  Commercial-Off-The-Shelf 
product. 

The  Technical  Assessment  was  conducted  from  May  through  September  1993. 
We  met  with  representatives  from  the  Aerospace  Guidance  and  Metrology 
Center;  the  Standard  Systems  Center;  and  the  Headquarters,  U.S.  Air  Force. 
We  also  discussed  the  PAMS  assessment  with  the  Air  Force  Director  of 
Metrology;  the  Air  Force  Material  Command's  Logistics  Applications  of 
Automated  Marking  and  Reading  Symbols  office;  a  Headquarters,  U.S.  Air 
Force,  computer  policy  representative;  and  the  Air  Force  Materiel  Command's 
PMEL  representative.  Additionally,  we  visited  a  PMEL  at  Andrews  Air  Force 
Base,  Maryland,  for  a  hands-on  demonstration  of  PAMS.  Appendix  C  lists  the 
organizations  that  we  visited  or  contacted. 


^Independent  verification  and  validation  is  a  technique  used  to  reduce  risk  in 
software  development.  An  independent  party  tests  the  software  and  ensures  that 
the  software  satisfies  all  system-level  requirements. 
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The  assessment  team  consisted  of  members  from  the  Technical  Assessment 
Division,  Audit  Planning  and  Technical  Support  Directorate.  The  team 
members  have  experience  in  computer  science,  engineering,  and  acquisition 
management. 
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Part  II  -  Findings  and  Reconunendations 
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Finding  A.  PAMS  Development 

The  Air  Force  had  not  wasted  resources  by  using  PAMS  instead  of  a 
Commercial-Off-The-Shelf  product.  Although  the  complainant 
identified  valid  PAMS  discrepancies,  the  Air  Force  also  recognized  these 
problems  and  corrected  them  for  the  most  part.  The  Air  Force, 
however,  has  yet  to  correct  one  PAMS  problem  that  stems  from  the 
deficiencies  identified  by  the  complainant  (See  Finding  B).  In  general, 
the  extent  of  the  PAMS  problems  have  not  been  as  significant  as  alleged 
by  the  complainant. 


Discussion 


At  the  time  of  the  allegation,  the  complainant  had  identified  valid  PAMS 
discrepancies.  PAMS  was  not  developed  in  accordance  with  Air  Force 
directives,  the  Air  Force  had  plans  to  reverse-engineer  PAMS  and  convert  the 
software  to  Ada,  and  the  Air  Force  had  not  conducted  an  independent  validation 
and  verification  of  PAMS. 

However,  the  Air  Force  also  recognized  the  PAMS  discrepancies  and,  in 
general,  has  corrected  them.  At  the  time  of  our  assessment,  the  status  of  the 
problems  cited  by  the  complainant  were  as  follows: 

o  the  Standard  Systems  Center  had  reviewed  the  PAMS  development 
documentation  for  compliance  with  Air  Force  regulations.  Some  documentation 
was  inadequate  and  was  being  updated.  In  general,  however,  the  Aerospace 
Guidance  and  Metrology  Center  and  the  Standard  Systems  Center  were 
managing  PAMS  in  accordance  with  the  applicable  directives  and  regulations 
(the  notable  exceptions  are  discussed  in  Finding  B); 

o  the  Air  Force  had  canceled  its  plans  to  reverse-engineer  PAMS  and 
convert  the  software  to  Ada;  and 

o  the  Standard  Systems  Center  validated  (tested)  the  PAMS  software 
when  the  PAMS  Program  Management  transferred  to  the  Standard  Systems 
Center.  Several  software  errors  were  identified  and  the  Standard  Systems 
Center  upgraded  the  software  accordingly.  The  revised  software  was  distributed 
to  the  PAMS  users  for  "operational"  validation  and  verification. 

The  complainant  believed  that  the  PAMS  discrepancies  would  have  two  major 
effects.  First,  the  PAMS  software  would  be  difficult  and  expensive  to  maintain. 
Second,  the  reverse-engineering  and  converting  of  the  PAMS  software  to  Ada 
would  be  extremely  costly.  Given  the  status  of  the  PAMS  system  at  the  time  of 
the  allegation,  the  complainant  was  right.  However,  as  a  result  of  the  Air 
Force's  subsequent  actions,  the  extent  of  the  PAMS  problems  have  not  been  as 
significant  as  contemplated  by  the  complainant. 
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Finding  A.  PAMS  Development 


At  the  time  of  the  allegation  (late  1992),  the  Standard  Systems  Center  had  just 
accepted  PAMS  program  management  responsibility  and  the  PAMS  software 
was  trouble-ridden  to  the  extent  that  some  PMELs  would  not  use  the  latest 
software  version.  Further,  the  PAMS  software  was  not  adequately  documented. 
These  factors  lead  the  complainant  to  conclude  that  the  PAMS  software  would 
be  difficult  and  costly  to  maintain.  However,  upon  accepting  programs 
management  responsibility  for  PAMS,  the  Standard  Systems  Center 
implemented  a  recovery  plan  and  was  able  to  develop  a  much  improved 
software  version  relatively  quickly,  without  exceeding  the  PAMS'  budger.  The 
recovery  plan  included  not  only  fixing  the  PAMS  software  errors  but  also 
updating  Ae  associated  documentation.  Consequently,  the  PAMS  software  has 
not  been  overly  difficult  or  expensive  to  maintain. 

In  a  similar  vein,  the  complainant's  allegation  that  reverse-engineering  PAMS 
and  converting  the  software  to  Ada  would  be  costly  was  correct.  However, 
significant  resources  were  never  expended  on  this  effort.  At  the  time  of  the 
allegation,  the  Standard  Systems  Center  had  initiated  a  project  to  reverse- 
engineer  and. convert  the  PAMS  software  to  Ada.  The  Standard  Systems  Center 
had  planned  to  use  the  Ada  software  on  other  Air  Force  systems.  Little  work 
was  actually  accomplished  on  this  project,  however,  before  the  Air  Force 
recognized  ftiat  the  project  was  not  cost-effective  and  cancelled  it. 

Further,  the  PAMS  users  were  satisfied  with  the  system.  Several  PAMS  users 
were  interviewed  and  all  indicated  a  general  satisfaction  with  PAMS 
performance.  Additionally,  these  users  felt  that  the  Standard  Systems  Center 
and  the  PAMS  Configuration  Control  Board  were  adequately  responding  to  the 
users'  requirements.  Although  some  users  indicated  that  PAMS  should  be 
upgraded  in  the  future,  none  stated  that  PAMS  should  necessarily  be  replaced 
with  a  Commercial-Off-The-Shelf  product.  In  short,  a  detailed  cost  analysis 
between  PAMS  and  the  Conunercial-Off-The-Shelf  products  would  need  to  be 
performed  to  justify  the  acquisition  of  a  new  system.  While  the  Standard 
Systems  Center  had  begun  to  research  possible  PAMS  upgrades,  the  PAMS 
communications  requirements  must  be  clearly  established  before  a  cost 
comparison  of  the  various  options  can  be  performed.  The  need  to  clearly  define 
the  PAMS  communications  requirements  is  further  discussed  in  Finding  B. 


Conclusion 


At  the  time  of  the  allegation,  the  complainant  cited  several  valid  PAMS 
problems,  which  could  have  resulted  in  waste.  However,  the  Air  Force  has 
corrected  these  problems  for  the  most  part.  Consequently,  resources  have  not 
been  wasted  on  PAMS  that  could  have  been  saved  by  replacing  PAMS  with  a 
Coimnercial-Off-The-Shelf  product. 


^The  Standard  Systems  Center  spent  approximately  $650,000  in  1992  and  1993 
on  PAMS. 
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Finding  B.  PAMS  Communications 

The  Aerospace  Guidance  and  Metrology  Center  routinely  did  not  receive 
TMDE  data  from  approximately  40  percent  of  the  Precision 
Measurement  Equipment  Laboratories.  This  condition  existed  because 
the  PAMS  communications^  requirements  and  management 
responsibilities  have  never  been  clearly  defmed.  A  short-term  solution 
to  the  PAMS  communications  problems  has  been  implemented,  but  the 
management  causes  have  not  been  addressed.  In  resolving  these 
problems,  the  Air  Force  will  alleviate  the  need  for  inefficient  short-term 
solutions,  avoid  the  higher  costs  of  correcting  the  problems  later,  and  lay 
a  solid  foundation  for  future  PAMS  enhancements. 


Discussion 


The  Aerospace  Guidance  and  Metrology  Center  routinely  does  not  receive 
TMDE  data  from  approximately  40  percent  of  the  Precision  Measurement 
Equipment  Laboratories  because  of  PAMS  communications  problems.  PAMS 
relies  on  other  Air  Force  maintenance  management  systems  to  transfer  TMDE 
data  from  the  Precision  Measurement  Equipment  Laboratories  to  the  Aerospace 
Guidance  and  Metrology  Center.  However,  PAMS  often  cannot  communicate 
with  these  other  systems.  The  details  on  the  PAMS  communications  problems 
are  in  Appendix  A,  but,  in  general,  PAMS  cannot  communicate  with  these  other 
systems  because  the  communications  requirements  (such  as  data  flow  padis  and 
interfaces)  between  the  systems  have  not  been  clearly  defined.  From  a 
management  perspective,  the  PAMS  communications  requirements  have  not 
been  clearly  defined  for  two  reasons: 

o  PAMS  was  not  adequately  integrated  into  a  Communications- 
Computer  Systems  (C-CS)  architecture  when  the  system  was  developed,  and 

o  a  PAMS  Program  Management  Directive  was  not  established  when 
the  PAMS  program  management  responsibility  transferred  from  the  PAMS 
Design  Office  to  the  Standard  Systems  Center. 

The  integration  of  PAMS  into  a  C-CS  architecture  and  the  establishment  of  a 
Program  Management  Directive  are  management  issues  that  result  in  technical 
problems.  The  Air  Force  has  recognized  ^s  fact  and  established  procedures  to 
prevent  the  PAMS-type  problems  from  occurring.  However,  these  procedures 
were  not  followed  when  PAMS  was  developed.  Specifically,  PAMS  was  not 
fully  integrated  into  a  C-CS  architecture  (that  is,  PAMS  interface  and  data  flow 
requirements  were  not  defmed),  contrary  to  Air  Force  Regulation  700-2, 


5ln  this  report,  communications  means  the  transfer  of  information  from  PAMS 
to  another  computer  system,  regardless  of  means  (e.g.,  modem  or  computer 
discs). 
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"Communications-Computer  Systems  Planning  and  Architectures."  In  general. 
Air  Force  Regulation  700-2  directs  that  all  Automated  Data  Processing  (ADP) 
systems  be  integrated  into  C-CS  architectures  to  minimize  interface  problems. 
The  purpose  of  integrating  ADP  systems  in  C-CS  architectures  is  more  fully 
explained  in  Air  Force  Pamphlet  700-50,  Volume  I,  Air  Force 
Communications-Computer  Systems  Architecture,  which  states: 

The  architectures  make  sure  validated  communications-computer 
systems  requirements  are  satisfied  with  integrated,  affordable 
technical  solutions.  Their  purpose  is  to  provide  standards,  systems, 
protocols,  interfaces,  and  so  forth,  that  must  be  considered  when 
developing,  implementing,  or  modifying  Air  Force  communications- 
computer  systems. 

PAMS  was  not  fully  integrated  into  a  C-CS  architecture  for  a  variety  of  reasons. 
Among  the  significant  reasons  were: 

o  the  Air  Force  wanted  to  field  PAMS  as  expeditiously  as  possible, 

o  several  computer  systems  that  PAMS  was  to  communicate  with  were 
being  updated  or  replaced,  and 

o  the  responsibility  for  developing  the  necessary  interfaces  was  levied 
upon  the  non-PAMS  systems  management. 

The  net  result  was  that  PAMS  was  fielded  with  little  regard  for  interfaces  or 
integration  with  other  systems,  which  is  precisely  the  situation  that  Air  Force 
Regulation  700-2  was  intended  to  avoid. 

Additionally,  Air  Force  regulations  were  not  followed  when  the  PAMS  program 
management  responsibility  transferred  to  the  Standard  Systems  Center.  PAMS 
program  management  responsibility  transferred  from  the  PAMS  Design  Office 
to  the  Standard  Systems  Center  in  the  summer  of  1992,  but  a  PAMS  Program 
Management  Directive  was  not  implemented  as  reguired  by  Air  Force 
Regulation  700-53,  "Management  of  Standard  Systems."”  Although  the  lack  of 
a  Program  Management  Directive  did  not  directly  cause  the  PAMS 
communications  problems,  it  has  prevented  the  timely  correction  of  the 
problems.  To  illustrate  this  point,  a  review  of  some  objectives  of  Air  Force 
Regulation  700-53  is  necessary.  These  objectives  include: 

o  providing  a  review  process  for  standard  systems  at  the  appropriate 
levels  witWn  the  Air  Force, 

o  ensuring  involvement  of  all  appropriate  Air  Force  organizations  and 
other  agencies  early  in  the  planning  process,  and 


^Air  Force  Regulation  700-53  directs  that  Program  Management  Directives  be 
implemented  for  Standard  Systems;  PAMS  is  a  Standard  System  because  more 
than  one  major  Command  utilizes  the  system. 
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o  ensuring  that  standard  systems  meet  Air  Force  Communications- 
Computer  Systems  architectural  guidance. 

To  meet  these  objectives,  Air  Force  Regulation  700-53  requires  that  the  Air 
Force  Functional  Manager  (in  PAMS  case,  the  Aerospace  Guidance  and 
Metrology  Center)  develop  a  Program  Management  Directive.  In  general,  the 
purpose  of  a  Program  Management  Directive  is  to  designate  the  organizational 
responsibilities  for  the  maintenance  of  a  standard  system. 

As  of  the  time  of  this  assessment,  however,  a  PAMS  Program  Management 
Directive  had  not  been  established.  Consequently,  the  corrective  actions  for  the 
PAMS  communications  problems  had  not  been  funded.  In  general,  the 
responsibilities  for  ensuring  that  one  computer  system  can  interface  with  another 
system  must  be  clearly  defined  so  that  the  proponents  of  each  system  can 
properly  plan  and  fund  for  changes  to  the  interface.  In  PAMS  case,  these 
responsibilities  have  not  been  defined.  As  a  result,  several  Air  Force 
maintenance  systems  have  made  changes  that  impacted  their  interface  with 
PAMS  but  no  funds  have  been  budgeted  to  correct  the  communications 
problems. 

While  Air  Force  Regulations  700-2  and  700-53  should  have  been  implemented 
earlier  in  PAMS  development,  three  factors  warrant  that  the  Air  Force 
"retroactively"  implement  these  regulations.  These  factors  are: 

o  the  PAMS  communications  capabilities  have  continued  to  deteriorate, 
forcing  the  Aerospace  Guidance  and  Metrology  Center  to  impose  an  additional 
person-power  burden  on  the  Precision  Measurement  Equipment  Laboratories; 

o  the  longer  the  Air  Force  delays  fixing  these  problems,  the  higher  the 
cost  of  implementing  the  fixes;  and 

0  until  the  Air  Force  corrects  the  PAMS  communications  problems,  the 
Air  Force  will  not  have  a  solid  foundation  upon  which  to  base  future  PAMS 
improvement  decisions. 

The  most  pressing  reason  for  fully  incorporating  PAMS  into  a  C-CS 
architecture  is  that  the  PAMS  communications  capabilities  continue  to 
deteriorate.  Since  being  fielded,  PAMS  has  been  able  to  communicate  with 
other  Air  Force  systems  to  varying  degrees.  Some  problems  have  been  site- 
specific;  but,  in  general,  PAMS  communications  have  continued  to  deteriorate 
because  changes  have  been  made  to  other  maintenance  management  systems 
without  regard  to  the  PAMS  communications  requirements.  An  example  best 
illustrates  this  point.  On  at  least  one  occasion,  the  Core  Automated 
Maintenance  System  (CAMS)  software  was  upgraded  but  the  PAMS  community 
was  not  informed  of  the  upgrade  until  after  the  changes  had  been  made.  As  a 
result  of  these  changes,  PAMS  was  not  able  to  communicate  with  CAMS.  The 
problem  was  fixed  only  after  the  PAMS  technical  representatives  met  with 
representatives  from  the  CAMS  community.  In  this  particular  case,  the 
problem  was  simple  so  significant  resources  were  not  required  to  remedy  the 
situation.  However,  such  is  not  always  the  case.  For  instance,  at  the  time  of 


10 


Finding  B.  PAMS  Conununications 


the  assessment,  another  CAMS  upgrade  was  planned  that  would  again  prevent 
PAMS  from  effectively  communicating  with  CAMS.  TTiis  problem  involves  the 
type  of  communications  lines  CAMS  uses  and  had  yet  to  be  resolved. 

Because  many  PAMS  communications  problems  have  not  been  easily  corrected, 
the  Air  Force  has  implemented  a  short-term  solution.  At  the  time  of  this 
assessment,  the  PAMS  community  had  been  trying  to  correct  the  PAMS 
communications  problems  for  at  least  a  year  with  little  success.  Consequently, 
in  May  1993,  the  PMELs  were  instructed  to  mail  their  TMDE  data  to  the 
Aerospace  Guidance  and  Metrology  Center  on  computer  discs.  While  this 
approach  is  expected  to  improve  the  percentage  of  TMDE  data  that  is  received 
at  the  Aerospace  Guidance  and  Metrology  Center,  it  imposes  a  greater  person- 
power  burden  on  the  PMELs  and  is  an  inefficient  use  of  Air  Force  computer 
resources. 

Given  the  circumstances,  the  Air  Force's  solution  makes  sense  for  the  short¬ 
term.  Nonetheless,  a  long-term  solution  needs  to  be  implemented,  and  the 
sooner,  the  better,  because  the  sooner  that  the  PAMS  communications  problems 
are  corrected,  the  lower  the  overall  cost.  In  general,  correcting  the  PAMS 
conununications  problems  will  entail  software  modifications.  While  the  cost  of 
developing  the  necessary  modifications  should  not  increase  significantly  over 
time,  die  overall  cost  of  the  fixes  can  be  expected  to  increase.  This  increase 
will  result  because  the  other  Air  Force  maintenance  management  systems  can  be 
expected  to  continue  evolving  without  regard  to  PAMS  communications 
requirements.  In  these  instances,  the  changes  may  need  to  be  re-worked  or 
additional  modifications  may  be  required  to  support  PAMS  requirements,  thus 
raising  the  overall  cost.  While  the  extent  of  these  types  of  problems  cannot  be 
forecasted,  the  historical  evidence  of  similar  problems  indicates  fairly  strongly 
that  these  types  of  problems  will  continue  in  the  absence  of  Air  Force  action. 

Another  reason  that  the  Air  Force  should  correct  the  PAMS  problems  is  that,  by 
doing  so,  the  Air  Force  will  lay  a  solid  foundation  for  future  PAMS 
improvements.  By  integrating  PAMS  into  a  C-CS  architecture  and 
implementing  a  PAMS  Program  Management  Directive,  the  Air  Force  will 
formalize  die  PAMS  communications  requirements  and  management 
responsibilities.  This  information  will  allow  Ae  Air  Force  to  rhore  readily 
upgrade  PAMS  in  the  future,  whether  the  improvement  be  a  simple 
enhancement  to  the  current  PAMS  configuration  or  a  complete  replacement  of 
the  system  with  a  Commercial-Off-The-Shelf  product.  Several  improvement 
options  are  discussed  in  Appendix  B. 


Conclusion 


Because  PAMS  cannot  communicate  with  other  Air  Force  maintenance 
management  systems,  the  TMDE  data  from  approximately  40  percent  of  the 
PMELs  routinely  does  not  reach  the  Aerospace  Guidance  and  Metrology 
Center.  The  communications  problems  were  caused  by  the  lack  of  program 
management  structure.  The  requirements  for  communications  had  never  been 
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explicitly  defined  and  no  organization  was  designated  to  correct  the  problems. 
Although  the  Air  Force  designed  a  short-term  solution,  that  solution  did  not 
address  the  underlying  cause  of  the  problems.  By  correcting  these  deficiencies, 
the  Air  Force  will  not  need  inefficient  short-term  solutions,  will  pay  less  for  the 
fixes,  and  will  better  support  future  PAMS  system  enhancements. 


Recommendations  for  Corrective  Action 


1.  We  recommend  that  Chairman  of  the  Headquarters,  U.S.  Air  Force 
-  Communications-Computer  Systems  Requirements  Board,  ensure  that  the 

Precision  Measurement  Equipment  Laboratory  Automated  Management 
System  communications  requirements  are  clearly  defined  by  fully 
incorporating  the  system  into  an  Air  Force  Communications-Computer 
Systems  arcMtecture  for  automated  maintenance  management  systems  in 
accordance  with  Air  Force  Regulation  700-2,  "Communications-Computer 
Systems  Planning  and  Architectures." 

Management  Comments.  The  Air  Force  agreed  with  the  substance  of  the 
recommendation  and  provided  an  alternative  approach  for  implementing  the 
recommendation.  The  Air  Force  stated  it  was  currently  evaluating  the 
migration  of  PAMS  and  other  maintenance  systems  into  a  standard  system 
called  the  Integrated  Maintenance  Data  System  (IMDS).  As  a  part  of  this 
process,  the  Air  Force  intends  to  clearly  define  the  PAMS  functionality  and 
communications  requirements.  The  IMDS  proof  of  concept  studies  will  be 
performed  in  FY  1995  and  a  competitive  acquisition  strategy  will  be  formulated 
for  solicitation  in  FY  1996.  The  complete  text  of  management  comments  is  in 
Part  IV. 

Technical  Assessment  Response.  The  Air  Force's  comments  were  responsive 
and  provided  a  reasonable  approach  for  implementing  our  recommendation. 
While  the  Air  Force's  comments  implied  that  IMDS  will  correct  the  PAMS 
communications  problems,  we  could  not  evaluate  this  position  as  IMDS  has  not 
yet  been  developed.  Whether  IMDS  is  developed  or  not,  the  Chairman  of  the 
Headquarters,  U.S.  Air  Force  Communications-Computer  Systems 
Requirements  Board,  will  need  to  integrate  PAMS  communications 
requirements  into  a  Communications-Computer  architecture  with  other  Air 
Force  maintenance  systems.  Consequently,  we  did  not  change  our  original 
recommendation. 

2.  We  recommend  that  the  Director,  Aerospace  Guidance  and  Metrology 
Center,  establish  overall  Precision  Measurement  Equipment  Laboratory 
Automated  Management  System  management  responsibilities  by 
implementing  a  corresponding  Program  Management  Directive  in 
accordance  with  Air  Force  Regulation  700-53,  "Management  of  Standard 
Systems." 
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Management  Comments.  The  Air  Force  agreed  with  the  substance  of  the 
recommendation,  stating  that  an  IMDS  Program  Management  Directive  is 
currently  being  developed.  The  Air  Force  intends  to  include  the  required 
PAMS  management  actions  in  this  Program  Management  Directive. 

Technical  Assessment  Response.  The  Air  Force's  comments  were  responsive 
and  provided  a  reasonable  approach  for  implementing  the  recommendation. 
Provided  that  IMDS  comes  to  fruition  as  planned,  a  separate  PAMS  Program 
Management  Directive  is  not  necessary. 


13 


This  page  was  left  out  of  orignial  document 


K 


Part  III  -  Additional  Information 


Appendix  A.  PAMS  Communications  Paths 


No  direct  connectivity  exists  between  the  Precision  Measurement  Equipment 
Laboratories  (PMELs)  and  the  Aerospace  Guidance  and  Metrology  Center,  so 
the  TMDE  calibration  data  is  transferred  from  PAMS,  through  several  other  Air 
Force  computer  systems,  to  the  Air  Force  Materiel  Command.  Because  PAMS 
and  these  other  Air  Force  maintenance  management  systems  employ  different 
software  protocols,  they  must  be  "interfaced"  for  the  data  transfer  to  occur. 
Once  the  data  arrives  at  the  Air  Force  Materiel  Command,  it  is  stored  on 
magnetic  tapes  and  delivered  to  the  Aerospace  Guidance  and  Metrology  Center 
by  truck.  This  data  flow  path  is  convoluted  and  complex;  and  routinely,  the 
Test,  Measurement,  and  Diagnostic  Equipment  (TMDE)  data  from 
approximately  40  percent  of  the  PMELs  does  not  reach  the  Aerospace  Guidance 
and  Metrology  Center. 

Depending  on  where  the  PMEL  is  located,  PAMS  interfaces  with  different 
systems.  For  most  PMELs,  PAMS  interfaces  with  the  Core  Automated 
Maintenance  System  (CAMS).  However,  if  the  PMEL  is  located  at  an  Air 
Logistics  Center,  PAMS  interfaces  with  a  subsystem  of  the  Depot  Maintenance 
Data  Systems  Network,  commonly  referred  to  as  "G004I."  CJ004I  is  not  a 
computer  system  in  the  classical  sense,  but  rather  a  combination  of  various 
maintenance  management  systems.  CAMS  and  G004I  are  mutually  exclusive 
systems,  but  both  are  connected  to  another  computer  system,  the  Product 
Performance  System,  designated  as  "D056."  PAMS  information  from  CAMS 
and  G004I  are  transferred  to  D056;  a  magnetic  tape  of  the  D056  output  is 
delivered  to  the  Aerospace  Guidance  and  Metrology  Center  by  truck.  Figure  1 
on  the  following  page  is  a  simplified  presentation  of  the  TMDE  data  flow  paths. 

Some  PAMS  interfaces  have  been  formally  established;  others  have  not.  The 
PAMS  to  CAMS  and  CAMS  to  D056  interfaces  have  been  formally  established 
in  Memorandums  of  Agreement,  but  the  data  transfer  process  is  still  error  prone 
and  manual  "massaging"  of  the  ^ta  is  often  required.  Additionally,  D056  may 
soon  be  replaced  by  the  Reliability  and  Maintainability  Information  System 
(REMIS);  however,  REMIS  is  not  able  to  process  PAMS  data.  No  interface 
agreement  has  been  established  between  PAMS  and  G004I  because  each  Air 
logistics  Center's  G004I  configuration  is  unique.  Some  Air  Logistics  Centers’ 
PAMS  data  does  reach  the  Aerospace  Guidance  and  Metrology  Center, 
however.  Like  D056,  G004I  may  soon  be  replaced  by  yet  another  computer 
system,  namely  the  Depot  Maintenance  Management  Information  System. 
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AGMC:  Aerospace  Guidonce  ond 
Metrology  Center 
ALC:  Air  Logistics  Center 

TMDE:  Test,  Measurement,  and 
Diagnostic  Equipment 

Figure  1.  TMDE  Doto  Flow  Paths 


ALC  PME 


Appendix  B.  Options  for  Improving  PAMS 


Several  options  exist  for  improving  PAMS'  capabilities,  but  recommendations 
concerning  PAMS  improvements  would  be  premature  before  the  Air  Force  has 
addressed  the  PAMS  communications  problems.  Once  the  PAMS 
communications  problems  have  been  resolved,  a  market  survey  can  be 
performed  and  the  costs  of  the  various  alternatives  compared  to  the  cost  of 
maintaining  PAMS.  In  the  interim,  maintaining  the  current  system  is  a 
reasonable  option.  The  following  discussion  on  upgrading  PAMS  is  based  on 
information  that  was  provided  by  the  complainant  and  the  Air  Force  and  is 
provided  for  information  purposes  only. 

The  options  for  upgrading  PAMS  that  were  reviewed  are  upgrading  the  PAMS 
hardware  and  so^are  or  replacing  PAMS  with  a  Commercial-Off-The-Shelf 
product.  Both  options  are  technically  feasible.  Other  options,  such  as  replacing 
PAMS  with  another  Air  Force  maintenance  management  system  or  connecting 
the  PMELs  directly  to  the  Aerospace  Guidance  and  Metrology  Center  might 
also  be  feasible. 

Upgrading  the  hardware  (and  to  some  aspects,  the  software)  of  the  PAMS  is 
feasible.  The  PAMS  software  can  only  be  run  on  UNISYS®  compatible 
computers  due  to  the  unique  operating  system;  however,  the  processing 
(throughput)  capabilities  can  be  readily  improved.  Additionally,  newer 
UNISYS  computers  can  run  MS-DOS®  software  applications. 

Replacing  PAMS  with  a  Commercial-Off-The-Shelf  product  is  also  possible. 
The  Standard  Systems  Center  has  reviewed  several  Commercial-Off-'nie-Shelf 
products  that  are  similar  to  PAMS  and  could  replace  the  system.  The  products 
vary  in  capabilities  and  may  require  replacement  of  the  PAMS  hardware  at  a 
significant  cost.  The  new  software  would  also  have  to  be  procured  and 
maintained.  Additionally,  the  Commercial-Off-The-Shelf  products  would 
require  modifications  to  be  compatible  with  the  existing  Air  Force  maintenance 
systems;  this  simation  is  essentially  the  same  as  exists  with  PAMS. 

Replacing  PAMS  with  another  Air  Force  maintenance  management  system 
might  be  a  viable  alternative  for  improving  PAMS.  Essentially,  PAMS  and 
other  Air  Force  maintenance  management  systems  accomplish  the  same 
functions  (database  management  of  maintenance  information)  and  should  have 
common  requirements.  Another  theoretically  possible  option  is  to  connect  the 
Precision  Measurement  Equipment  Laboratories  directly  to  the  Aerospace 
Guidance  and  Metrology  Center  with  a  wide-area  network. 

Trademarks 


UNISYS  is  a  registered  trademark  name  for  UNISYS  Corporation,  San  Jose,  California 

MS-DOS  is  a  registered  trademark  name  for  Microsoft  Disc  Operating  System,  Microsoft 
Corporation,  ReSmond,  Washington 
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Department  of  the  Air  Force 

Headquarters,  U.S.  Air  Force,  Pentagon,  Washington,  DC 
Air  Force  Materiel  Command,  Wright-Patterson  Air  Force  Base,  OH 
Aerospace  Guidance  and  Metrology  Center,  Newark  Air  Force  Base,  OH 
Standard  Systems  Center,  Gunter  Annex,  Maxwell  Air  Force  Base,  AL 
89th  Equipment  Maintenance  Squadron,  Andrews  Air  Force  Base,  MD 
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Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Auditor  General,  Department  of  the  Air  Force 
Aerospace  Guidance  and  Metrology  Center 
Standard  Systems  Center 


Other  Defense  Organizations 

Director,  Defense  Information  Systems  Agency 

Director,  Defense  Logistics  Agency 

Director,  National  Security  Agency 

Inspector  General,  Central  Imagery  Office 

Inspector  General,  Defense  Intelligence  Agency 

Inspector  General,  National  Security  Agency 

Director,  Defense  Logistics  Studies  Information  Exchange 


Non-Defense  Federal  Organizations 

Office  of  Management  and  Budget 

Technical  Information  Center,  National  Security  and  International  Affairs  Division, 
General  Accounting  Office 

Chairman  and  Ranking  Minority  Member  of  each  of  the  following  Congressional 
Committees  and  Subcommittees: 

Senate  Subcommittee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 

Senate  Committee  on  Armed  Services 

Senate  Committee  on  Governmental  Affairs 

House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Operations 

House  Subcommittee  on  Legislative  and  National  Security,  Committee  on 
Government  Operations 
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Department  of  the  Air  Force  Comments 


DEPARTMENT  OF  THE  AIR  FORCE 

MtADOUAHTtHS  UNITED  STATES  AIN  PONCE 


U  1WI 

MEMCHIANDUM  PC»  ASSISTANT  INSPECTW  GENERAL  TOR  AUDITING. 

OFFICE  OT  THE  INSPECTOR  GENERAL. 

MPARTMENT  OF  I«FENSE 

FROM:  HQUSAF/LGM 

1030  Air  Face  Pentagon 
W«shin|toa  D.C.  20330-1030 

SUBJECT;  Draft  Technic*]  Assessment  Rqwrt  of  AePiedaonMeasuiemeoiEquipinem 

Laboratory  Automated  Management  Systtra  (PAMS)  (Projea  No.  3PT-8016).  13  June  94 

This  is  in  reply  to  your  request  for  Air  Force  comments  on  the  subjea  draft  report.  We 

concur  in  pan  with  the  recommendations  to  Findinj  B.  Hie  Air  Force  is  cutrenOy  evaluating 

migrating  the  PAMS  and  other  maintenance  dau  systems  into  a  standard  core  lysttm  called  the 

Integrated  Maintenance  Data  System  (IMDS).  Tliis  system  is  envisioned  to  be  a  aeamless 
maintenance  data  system  integrating  numerous  mainicnana  processes.  A  program  management 
direcuve  (PMD)  is  currenUy  being  developed  fa  the  IMDS  and  it  is  our  intent  to  include  tte  reouired 
PAMS  manageraem  actions  in  ihis  PMD. 


The  IMDS  System  Program  Oftiu  is  currently  being  formed  within  the  Electronic  Systems 
center .  This  program  office  will  perform  fimctiooal  economic  analyses  to  determine  core  process 
requuements  fa  the  IMDS.  As  a  result  of  these  analyses.  PAMS  functionality  may  be  moved  into 
the  core  process  a  evolved  into  *  system  module  at  which  time  communications  requirements  will  be 
clearly  defined  IMDS  proof  of  concept  studies  will  be  performed  in  Fy95  and  a  competitive 
acquisttion  strategy  will  be  formulated  for  solicitation  in  FY96.  This  IMDS  effort  has  Air  Staff 
Office  of  the  Secretary  of  Defense,  and  Congressional  suppoa 

Questions  may  be  directed  to  Mr  Tom  Ght.  HQ  USAF/LGMM.  DSN  227-3859. 
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